System and method for image localization of effecters during a medical procedure

ABSTRACT

A computer-assisted imaging and localization system assists the physician in positioning implants and instruments into a patient&#39;s body. The system displays overlapping images—one image of the surgical site with the patient&#39;s anatomy and another image showing the implant(s) or instrument(s). The overlapping image of the implant/instrument is moved over the static image of the anatomy as the implant/instrument is moved. These moving image of the implant/instrument can be an unaltered image or an image altered to intensify or mitigate the anatomical or non-anatomical aspects of the moving image. Sliding these images over one another helps the surgeon in positioning devices or instruments with a high degree of accuracy and with a limited number of additional x-rays.

PRIORITY CLAIM AND REFERENCE TO RELATED APPLICATIONS

This application is a utility filing from and claims priority toco-pending U.S. Provisional Application No. 62/336,999, entitled “Systemand Method for Image Localization of Effecters During a MedicalProcedure” filed on May 16, 2016, the entire disclosure of which isincorporated herein by reference, and to U.S. Provisional ApplicationNo. 62/337,010, also filed on May 16, 2006, the disclosure of which isincorporated herein by reference. This application is also a utilityfiling from and claims priority to co-pending U.S. ProvisionalApplication No. 62/374,187, entitled “Detection of Tracked Radio-denseobjects During Imaging”, filed on Aug. 12, 2016, the entire disclosureof which is incorporated herein by reference.

BACKGROUND

Many surgical procedures require obtaining an image of the patient'sinternal body structure, such as organs and bones. In some procedures,the surgery is accomplished with the assistance of periodic images ofthe surgical site. Surgery can broadly mean any invasive testing orintervention performed by medical personnel, such as surgeons,interventional radiologists, cardiologists, pain management physicians,and the like. In surgeries and interventions that are in effect guidedby serial imaging, which we will refer to as image guided, frequentpatient images are necessary for the physician's proper placement ofsurgical instruments, be they catheters, needles, instruments orimplants, or performance of certain medical procedures. Fluoroscopy, orfluoro, is one form of intraoperative X-ray and is taken by a fluorounit, also known as a C-arm. The C-arm sends X-ray beams through apatient and takes a picture of the anatomy in that area, such asskeletal and vascular structure. It is, like any picture, atwo-dimensional (2D) image of a three-dimensional (3D) space. However,like any picture taken with a camera, key 3D info may be present in the2D image based on what is in front of what and how big one thing isrelative to another.

A DRR is a digital representation of an X-ray made by taking a CT scanof a patient and simulating taking X-rays from different angles anddistances. The result is that any possible X-ray that could be acquiredfor that patient can be simulated, which is unique and specific to howthe patient's anatomical features look relative to one another. Becausethe “scene” is controlled, namely by controlling the virtual location ofa C-Arm to the patient and the angle relative to one another, a picturecan be generated that should look like any X-ray taken in the operatingroom (OR).

Many imaging approaches, such as taking fluoro images, involve exposingthe patient to radiation, albeit in small doses. However, in these imageguided procedures, the number of small doses adds up so that the totalradiation exposure can be problematic not only to the patient but alsoto the surgeon or radiologist and others participating in the surgicalprocedure. There are various known ways to decrease the amount ofradiation exposure for a patient/surgeon when an image is taken, butthese approaches come at the cost of decreasing the resolution of theimage being obtained. For example, certain approaches use pulsed imagingas opposed to standard imaging, while other approaches involve manuallyaltering the exposure time or intensity. Narrowing the field of view canpotentially also decrease the area of radiation exposure and itsquantity (as well as alter the amount of radiation “scatter”) but againat the cost of lessening the information available to the surgeon whenmaking a medical decision. Collimators are available that can speciallyreduce the area of exposure to a selectable region. For instance, acollimator, such as the Model Series CM-1000 of Heustis Medical, isplaced in front of an x-ray source, such as the source 104 shown inFIG. 1. The collimator consists of a series of plates that absorb mostincident X-rays, such as lead. The only x-rays that reach the patientare those that pass through apertures between the plates. The positionof the plates can be controlled manually or automatically, and theplates may be configured to provide differently shaped fields, such amulti-sided field. Since the collimator specifically excludes certainareas of the patient from exposure to x-rays, no image is available inthose areas. The medical personnel thus have an incomplete view of thepatient, limited to the specifically selected area. Thus, while the useof a collimator reduces the radiation exposure to the patient, it comesat a cost of reducing the amount of information available to the medicalpersonnel.

A typical imaging system 100 is shown in FIG. 1. The imaging systemincludes a base unit 102 supporting a C-arm imaging device 103. TheC-arm includes a radiation source 104 that is positioned beneath thepatient P and that directs a radiation beam upward to the receiver 105.It is known that the radiation beam emanated from the source 104 isconical so that the field of exposure may be varied by moving the sourcecloser to or away from the patient. The source 104 may include acollimator that is configured to restrict the field of exposure. TheC-arm 103 may be rotated about the patient P in the direction of thearrow 108 for different viewing angles of the surgical site. In someinstances, metal or radio-dense material effecters, such as implants orinstruments T, may be situated at the surgical site, necessitating achange in viewing angle for an unobstructed view of the site. Thus, theposition of the receiver relative to the patient, and more particularlyrelative to the surgical site of interest, may change during a procedureas needed by the surgeon or radiologist. Consequently, the receiver 105may include a tracking target 106 mounted thereto that allows trackingof the position of the C-arm using a tracking device 130. For instance,the tracking target 106 may include several infrared emitters spacedaround the target, while the tracking device is configured totriangulate the position of the receiver 105 from the infrared signalsemitted by the element. The base unit 102 includes a control panel 110through which a radiology technician can control the location of theC-arm, as well as the radiation exposure. A typical control panel 110thus permits the technician to “shoot a picture” of the surgical site atthe surgeon's direction, control the radiation dose, and initiate aradiation pulse image.

The receiver 105 of the C-arm 103 transmits image data to an imageprocessing device 122. The image processing device can include a digitalmemory associated therewith and a processor for executing digital andsoftware instructions. The image processing device may also incorporatea frame grabber that uses frame grabber technology to create a digitalimage or pixel-based image for projection as displays 123, 124 on adisplay device 126. The displays are positioned for interactive viewingby the surgeon during the procedure. The two displays may be used toshow images from two views, such as lateral and AP, or may show abaseline scan and a current scan of the surgical site. An input device125, such as a keyboard or a touch screen, can allow the surgeon toselect and manipulate the on-screen images. It is understood that theinput device may incorporate an array of keys or touch screen iconscorresponding to the various tasks and features implemented by the imageprocessing device 122. The image processing device includes a processorthat converts the image data obtained from the receiver 105 into adigital format. In some cases the C-arm may be operating in thecinematic exposure mode and generating many images each second. In thesecases, multiple images can be averaged together over a short time periodinto a single image to reduce motion artifacts and noise.

Standard X-ray guided surgery typically involves repeated x-rays of thesame or similar anatomy as an effecter (e.g.—screw, cannula, guidewire,instrument, etc.) is advanced into the body. This process of moving theeffecter and imaging is repeated until the desired location of theinstrument is achieved. This iterative process alone can increase thelifetime risk of cancer to the patient over 1% after a single x-rayintensive intervention.

Classic image guided surgery (“IGS”) uses prior imaging as a roadmap andprojects a virtual representation of the effecter onto virtualrepresentations of the anatomy. As the instrument is moved through thebody, the representation of the effecter is displayed on a computermonitor to aid in this positioning. The goal is to eliminate the needfor x-rays. Unfortunately, in practice, the reality of these devicesdoesn't live up to the desire. They typically take significant time toset-up, which not only limits adoption but only makes them impracticalfor longer surgeries. They become increasingly inaccurate over time asdrift and patient motion cause a disassociation between physical spaceand virtual space. Typical IGS techniques often alter work flow in asignificant manner and do not offer the physician the ability to confirmwhat is occurring in real-time and to adjust the instrument as needed,which is a primary reason fluoroscopy is used.

What would benefit greatly the medical community is a simple imagelocalizer system that helps to position instruments without alteringworkflow. It would be substantially beneficial if the system can quicklybe set-up and run, making it practical for all types of medicalinterventions both quick and protracted. The desirable system wouldsignificantly limit the number of x-rays taken, but does not requireeliminating them. Therefore, by both encouraging reimaging and usingthis as a means to recalibrate, the system would ensure that theprocedure progresses as planned and desired. Using the actual x-rayrepresentation of the effecter rather than a virtual representation ofit would further increase accuracy and minimize the need for humaninteraction with the computer. If the system mimics live fluoroscopybetween images, it would help to position instruments and provide theaccuracy of live imaging without the substantial radiation imparted byit.

SUMMARY OF THE DISCLOSURE

A computer-assisted imaging localization system is provided that assiststhe physician in positioning implants and instruments into a patient'sbody. The system has the desired effect of displaying the actualinstrument or implant and using this displayed to guide surgery withoutthe need to directly interact with the computer. The system does so bydisplaying and moving overlapping images on a computer screen, allowingone image to be seen through the other. These image “masks” can be theunaltered image or doctored images to intensify or mitigate theanatomical or non-anatomical aspects of the image. Sliding these imagesover one another can help to position medical devices with a high degreeof accuracy with a limited number of additional x-rays.

DESCRIPTION OF THE DRAWINGS

FIG. 1 is a pictorial view of an image guided surgical setting includingan imaging system, an image processing device and a localizer ortracking device for surgical instruments and devices.

FIG. 2 is a diagram of steps in displaying movement of a trackedeffecter on an x-ray image of a surgical site.

FIGS. 3A-D are screen shots of image displays of a surgical site showingthe patient's anatomy and a movable image of a radio-dense effecter inrelation to a fixed image of the surgical site.

FIGS. 4A-C are screen shots of x-ray images of a surgical site andradio-dense effecter, including a low dose x-ray image and an image inwhich the display of the radio-dense effecter is enhanced relative tothe image of the anatomy.

FIGS. 5A-C are screen shots of x-ray images in which the radio-denseeffecter is represented by a metal mask in an image that moves relativeto the fixed image of the surgical site as the effecter moves.

FIGS. 6A-B are screen shots of x-ray images of the surgical site with anoverlaying metal mask image of the effecter.

FIG. 7 is a screen shot of an x-ray image with slugs indicating theposition of the tip of radio-dense effecters relative to the anatomyshown in the image.

FIG. 8 is a side view of a generic effecter having marker bands used fortracking the position of the effecter.

FIG. 9 is a side view of a generic effecter having a tracking elementmounted on the effecter and providing marker bands for tracking theposition of the effecter.

FIG. 10 is a side view of a generic effecter having another trackingelement mounted on the effecter and providing marker bands for trackingthe position of the effecter.

FIG. 11 is screen shot of an x-ray image of a surgical field with aneffecter and a region of interest within the viewing field.

FIGS. 12A-C are screen shots of low dose x-ray images showing images ofradio-dense effecters.

FIGS. 13A-F are screen shots of x-ray images of multiple radio-denseeffecters in a surgical field with images of the effecters isolated andrepresented by metal masks overlaid onto the image of the anatomy.

FIGS. 14A-E are a series of screen shots of an x-ray image in which theradio-dense effecters are automatically detected by the image processingdevice of the present disclosure.

FIG. 15A is a representation of a movement of the x-ray device or c-armduring a surgical procedure.

FIGS. 15B-D are screen shots of an x-ray image showing the movement ofthe image corresponding to the movement of the c-arm in FIG. 15A.

FIG. 16A is a representation of a movement of a radio-dense effecterduring a surgical procedure.

FIGS. 16B-C are screen shots of an x-ray image showing the movement ofthe image corresponding to the movement of the radio-dense effecter inFIG. 16A with the position of effecter remaining stationary.

FIG. 17A is a representation of a movement of a radio-dense effecterduring a surgical procedure.

FIGS. 17B-C are screen shots of an x-ray image showing the movement ofthe image corresponding to the movement of the radio-dense effecter inFIG. 16A with the position of the image of the anatomy remainingstationary.

FIG. 18A is a representation of a movement of a radio-dense effecterduring a surgical procedure.

FIGS. 18B-C are screen shots of an x-ray image showing the movement ofthe image corresponding to the movement of the radio-dense effecter inFIG. 18A with the position of effecter remaining stationary and withgrid lines superimposed on the image corresponding to the stationaryorientation of the effecter.

FIG. 19A is a representation of a movement of a radio-dense effecterduring a surgical procedure.

FIGS. 19B-C are screen shots of an x-ray image showing the movement ofthe image corresponding to the movement of the radio-dense effecter inFIG. 19A with the position of the image of the anatomy remainingstationary and with grid lines superimposed on the image correspondingto the different positions of the radio-dense effecter.

FIG. 20 are screen shots of x-ray images illustrating the low visibilityof certain radio-dense effecters in a surgical site.

FIG. 21 is a flow chart for detecting the presence and location of aradio-dense effecter in an image of a surgical site.

FIG. 22 is a screen shot of an x-ray image of an effecter in a surgicalsite illustrating one step of the detection method in the flow chart ofFIG. 21.

FIG. 23 is a screen shot of an x-ray image of an effecter in a surgicalsite illustrating a further step of the detection method in the flowchart of FIG. 21.

FIG. 24 is a screen shot of an x-ray image of an effecter in a surgicalsite illustrating another step of the detection method in the flow chartof FIG. 21.

FIG. 25 is a screen shot of an x-ray image of an effecter in a surgicalsite illustrating a subsequent step of the detection method in the flowchart of FIG. 21.

FIG. 26 is a screen shot of an x-ray image of an effecter in a surgicalsite illustrating yet another step of the detection method in the flowchart of FIG. 21.

FIG. 27 is a screen shot of an x-ray image of an effecter in a surgicalsite illustrating one step of the detection method in the flow chart ofFIG. 21.

FIG. 28 is a screen shot of an x-ray image of a surgical site in whichthe effecter has been detected and the metal mask of the effecterenhanced within the image.

FIG. 29A is an image of a surgical field acquired using a full dose ofradiation in the imaging system.

FIG. 29B is an image of the surgical field shown in FIG. 29A in whichthe image was acquired using a lower dose of radiation.

FIG. 29C is a merged image of the surgical field with the two imagesshown in FIGS. 29A-B merged in accordance with one aspect of the presentdisclosure.

FIG. 30 is a flowchart of graphics processing steps undertaken by theimage processing device shown in FIG. 1.

FIG. 31A is an image of a surgical field including an object blocking aportion of the anatomy.

FIG. 31B is an image of the surgical field shown in FIG. 31A with edgeenhancement.

FIGS. 31C-31J are images showing the surgical field of FIG. 31B withdifferent functions applied to determine the anatomic and non-anatomicfeatures in the view.

FIGS. 31K-31L are images of a mask generated using a threshold and atable lookup. FIGS. 31M-31N are images of the masks shown in FIGS.31K-31L respectively, after dilation and erosion.

FIGS. 31O-31P are images prepared by applying the masks of FIGS.31M-31N, respectively, to the filter image of FIG. 31B to eliminate thenon-anatomic features from the image.

FIG. 32A is an image of a surgical field including an object blocking aportion of the anatomy.

FIG. 32B is an image of the surgical field shown in FIG. 32A with theimage of FIG. 32A partially merged with a baseline image to display theblocked anatomy.

FIGS. 33A-B are displays of the surgical field adjusted for movement ofthe imaging device or C-arm and providing an indicator of alignment ofthe imaging device with a desired trajectory for acquiring a new image.

FIG. 34 is a graphical representation of an image alignment processaccording to the present disclosure.

FIG. 35A is an image of a surgical field obtained through a collimator.

FIG. 35B is an image of the surgical field shown in FIG. 35A as enhancedby the systems and methods disclosed herein.

DETAILED DESCRIPTION

For the purposes of promoting an understanding of the principles of thedisclosure, reference will now be made to the embodiments illustrated inthe drawings and described in the following written specification. It isunderstood that no limitation to the scope of the disclosure is therebyintended. It is further understood that the present disclosure includesany alterations and modifications to the illustrated embodiments andincludes further applications of the principles disclosed herein aswould normally occur to one skilled in the art to which this disclosurepertains.

According to one aspect of the invention, the process begins with takingan image of the anatomy to be addressed surgically. Typically this“localizing shot” or “baseline image” does not contain the radio-denseeffecter (e.g.—screw, cannula, guidewire, instrument, etc.) that is tobe moved/adjusted, although in one embodiment a single image containingthe effecter can be used. The image processing device 122 (FIG. 1)generates a digital image that can be displayed and manipulateddigitally. With the anatomy identified and displayed on a computerscreen, a “new” image with the effecter or instrument is taken, withthis image also converted to a digital image by the image processingdevice 122. This new image is displayed on top of the originallocalizing shot so that the resulting image looks like the conventionalimage on a fluoroscope screen, such as shown in FIG. 3A. In one aspectof the present disclosure, the effecter, such as effecter T in FIG. 1,incorporates a localizer system (e.g.—EM, Optical IGS, etc) capable oftracking movement of the effecter. The 3D movement of the effectermeasured by the localizer system can be applied to the digitalrepresentation of the “new” image relative to move the “new” imagerelative to the “localizing shot” image. Thus, as the tip of theeffecter is tracked, the movement of the “new” image shows the change inposition of the tip of the instrument being tracked relative to thestationary anatomy depicted in the “localizing shot”. On the computerscreen, it thus appears as if live fluoroscopy is being taken as theeffecter is being moved and as if the actual tool or implant is beingmoved and adjusted relative to the patient's anatomy. When the nextimage is taken, the tip of the effecter is at the location that thephysician desires. It can be appreciated that unlike the typical IGSsystem in which a digital model of the effecter is manipulated, thesystem and method of the present disclosure relies on manipulating anactual image of the effecter in the surgical field.

The movement of the “new” image on the display is based on the geometryof the tip of the effecter relative to the location within the cone beamof the fluoroscope, as depicted in FIG. 2. The nearer the tip of thetracked effecter is to the x-ray source, for the same relative movement,the greater the movement of the “new” image and therefore the effecter'sprojection (in pixels) relative to the size of the “localizing shot”.Assuming a standard size image, such as a 9 in. image intensifier, andassuming a typical 1000 mm separation of the x-ray source from theintensifier, there is an approximate 2.24 pixel per mm movement of thetracked effecter projected on the image intensifier. Away from the imageintensifier and closer to the source, this pixel-per-mm movement ratiois magnified in a consistent manner as shown in FIG. 2. In particular,the movement distance of the projection of the tracked effecter on theimage intensifier is given by Y′=X′ * Y/X, where Y is the actualmovement distance of the effecter, X is the distance from the source tothe tracked effecter/instrument, X′ is the distance from the source tothe localizing image at the image intensifier and Y′ is the projectedmovement distance. It can be appreciated that the distance X′ istypically fixed throughout the procedure for a conventional C-arm X-raysource. The distance X and the movement distance Y can be determined bythe image processing device 122 (FIG. 1) based on data received from thelocalizer system used to track the movement of the effecter. The imageprocessing device uses the projected movement distance Y′ to move the“new” image accordingly on the display.

The “new” image, shown in the lower representation in FIG. 2, can betaken using standard x-ray settings, or may be taken using less thanfull dose radiation or low dose settings which has the benefit ofblurring out the anatomy while having relatively little impact on theimage of a metal or radio-dense material effecter in the image. (It isunderstood that a “radio-dense” material generally does not allow theimaging rays or x-rays to pass through so that the radio-dense effecterblocks the underlying anatomy). When the “new” image is a low doseimage, the “new” image can be combined with or overlaid on the imagefrom the localizing shot allowing the user to see the resulting combinedimage with the appearance of the anatomy appearing as a livefluoroscopic image. The result is an image as seen in FIG. 3A that canhelp guide an effecter to the correct location desired by the physician.

In the example shown in FIGS. 3A-D, a bone screw 10 to be tracked isintroduced into a patient after an initial “localizing shot” andprojected on the display 122/123 (FIG. 1) as the screen shot of FIG. 3A.As the tracked instrument 10 is moved out of the field of the localizingshot or baseline image 12, as depicted in the screen shot of FIG. 3B,the two overlapping images can be appreciated, with the localizing shot12 seen to the left and the new low radiation image 14 to the right. Itcan be noted that the metal screw in the low radiation image is veryprominent while the representation of the anatomy is obscure. When thetracked screw is moved into an ideal location based on the desire of thephysician, such as shown in the screen shot of FIG. 3C, the image on thescreen can constantly project a combined image (overlaying the full doselocalizing shot with the low dose image) that replicates what a newfluoroscopic image would look like at any point, mimicking livefluoroscopy without obtaining a new live image. It can be appreciatedthat the localizing or baseline image 12 does not change as the effecter10 is moved, at least so long as the C-arm or X-ray source is not moved.Thus, the digital data for the localizing image 12 is not manipulated bythe image processing device during movement of the effecter. On theother hand, the image processing device does manipulate the digital dataof the “new” image based on the projected movement of the trackedeffecter so that the “new” image moves across the display as theeffecter is moved.

A stationary full dose new image can be taken, such as the display inthe screen shot of FIG. 3D, to confirm that the effecter 10 is in thelocation desired by the physician. If for some reason the imagealignment is off or further fine tuning is required, this newly acquiredimage can replace the prior localizing shot image as the baseline image,and the process is repeated. The system thus resets or recalibrates whenthe full dose new image is taken, so that subsequent images are alwaysmore accurately displayed than previous ones.

It can be appreciated that as the physician moves the effecter 10 thelow dose image moves with the effecter. When the effecter is within thefield of the baseline or localizing shot image, as in FIG. 3C, the imageof the effecter from the low dose image is combined with the stationarylocalizing image so that the physician can clearly see the patient'sanatomy and the effecter's position relative to that anatomy. As theeffecter is moved within the field of the baseline image, the image ofthe effecter (and the “new” image) moves accordingly so that thephysician can guide the tip of the effecter to the desired position inthe anatomy. In recognition that a new image is not actually beingacquired during each step of movement of the effecter, the physician canacquire new low dose images at various stages of movement of theeffecter to verify the actual location of the effecter. Thus, any errorin the actual vs. displayed position of the effecter relative to theanatomy is eliminated with each new low dose image taken. In otherwords, with each low dose image, the system recalibrates the actualposition of the effecter relative to the anatomy based on the digitaldata acquired from the low dose image. The new data identifying the newposition of the effecter is then the starting point for movement of thenew image as the effecter is moved by the surgeon. It is contemplatedthat the physician may require multiple low dose images as the effecteris moved into its final position, with each low dose image recalibratingthe actual position of the effecter, potentially culminating in a fulldose image to verify the final position.

In one aspect, each new low dose image can be processed according to thetechniques described U.S. Pat. No. 8,526,700 (the '700 Patent), whichissued on Sep. 3, 2013, the entire disclosure of which is incorporatedherein by reference. As described in more detail in the '700 Patent afull dose image is manipulated into a multitude of orientations, with animage of each of the orientations stored in memory. The low dose imageis compared to these multitude of stored images to find a “full dose”image that matches the current low dose image. The new low dose image isthen merged with the extracted full dose image to produce a display thatsimulates an actual full dose image. It can be appreciated that this newmerged image is only of the anatomy; however, the actual low dose imageshowing the effecter can be overlaid on the new “full dose” image, asdescribed above. The presence of the effecter in the low dose image usedto obtain the new merged image can be accounted for as described in the'700 Patent and in further detail herein.

Although a low radiation image is shown in FIGS. 3A-D, a conventional orfull dose “new” image can be taken and displayed with similar results,as shown in the screen shot of FIG. 4A. A low radiation image can beused, as see in the screen shot of FIG. 4B, or a metal intensificationof the “new” image can be performed as shown in the screen shot of FIG.4C. The image of FIG. 4B is obtained under low radiation so that theanatomic features are effectively washed out. While the image of theeffecter 10 is also washed out due to the low dosage, the meta or otherradio-dense material is sufficiently radiopaque so that the resultingimage of the effecter in FIG. 3B is still outstanding enough to beeasily seen.

The image of FIG. 4C is generated by intensifying the pixels associatedwith the image of the effecter 10, so that when the full image isdisplayed the image of the effecter essentially washes out the image ofthe underlying anatomy. In either case, what is projected has theability to “fool the eye” to make it appear to the surgeon as if theinstrument is moving under live fluoroscopy.

The metal intensification image of FIG. 4C can constitute a metal maskapplied to the images, such as the image in the screen shot of FIG. 5A.As shown in FIG. 5B, the image of the effecter 10 is represented by agreen mask 20 overlaying the actual image. The movement of the mask iscorrelated to the actual movement of the effecter as determined by thelocalizer. When the green layer of the mask 20 is moved to a more ideallocation, a confirmatory x-ray can be taken as in the screen shot ofFIG. 5C. The green or metal mask 20 can be generated by the imageprocessing device 122 (FIG. 1) using software that examines the pixelsof the image to determine which pixels are associated with anatomicfeatures and non anatomic features based primarily on the intensityvalue of each pixel. Various filters can be applied each pixel of thedigitized X-ray image to enhance the edges between pixels representinganatomic and non-anatomic features. Once the pixels associated with thenon-anatomic features are acquired and the edges enhanced, the pixelsoutside the selected non-anatomic pixels can be washed out, leaving onlythe pixels for the non-anatomic feature corresponding to the effecter.

Similar to the images of FIGS. 5A-C, image tracking can be applied inFIGS. 6A_B to a Jamshedi needle 10′ that is repositioned to a desiredposition in the patient's body. However, in FIG. 6A there is no initial“localizing shot”. The “new” image serves as both the stationary and themoved image. The image of the effecter is replaced by a green layermask, such as the mask 20 of FIG. 5C, and just the green layer of theimage is moved on the background of the “new” image. The image guidancesystem of the effecter can determine the relative location of theinstrument in the image, so that rather than moving the entire image asin the prior examples, only a narrow area around the region of theeffecter 10′ is moved.

The present invention contemplates a system and method for moving imagemasks or overlapping image sets based on the movement of a trackedobject, which provides the physician or surgeon with the ability toplace a surgical effecter at the correct location inside a patient witha minimal number of X-ray images. Movement projection is not based onthe absolute motion of the effecter but rather on the relative motion ofthe tracked effecter within the imaging space. Although knowledge of theabsolute location of the tip of the effecter is needed for certain imagemovements, such as shown in FIG. 6B, such knowledge is not necessary. Itis only necessary to know the relative motion between the originalposition and the new position of the effecter, and the distance from thetip of the effecter/instrument to the X-ray source.

The position of the effecter/instrument is recalibrated on each newX-ray shot. On the instrument side this means that each x-ray resets therelative position or the initial starting point of the “new” image tothe current location of the tracked effecter to which is linked a “new”image with that effecter in it. This feature makes the system mostlyfocused on relative movement so that the potential time horizon fordrift to set in is minimized.

The system and method disclosed herein creates “pseudo-livefluoroscopy”, meaning that the physician/surgeon can see the movement ofthe effecter/instrument in real-time without constant imaging of thepatient. The present disclosure further contemplates automating takingimages to create constantly re-updated spot images with “pseudo-livefluoroscopy” in between to create a continuous high accuracy instrumenttracking device with a live fluoroscopy appearance with dramaticallyfewer images and resulting radiation. The methods of the presentdisclosure only require knowledge of relative movement (meaning thedelta between the last position of the instrument to the current) andonly require displaying the 2D motion of the effecter/“new” image tomake this functional. The present disclosure provides a morecomprehensive imaging system compared to typical IGS where it isnecessary to know the absolute movement and the actual knowledge of whatis being moved (in order to project a correct virtual representation ofit).

The system and method of the present invention works with a metal maskor an actual image, and can work with low dose images or full doseimages. With this system, the entire image can be moved or adjusted, asshown in FIGS. 3, 4, or only a region of interest is moved or adjusted,as shown in FIG. 6B.

The system and method disclosed herein uses the actual effecter (or morespecifically an active x-ray picture of the effecter), not a virtualrepresentation of it as in a typical IGS. This approach makes itpossible to emphasize or deemphasize different features (e.g.—anatomy,metal, etc) of the two images to aid in visualization. The methodsdisclosed herein do not require distortion correction or dewarping, or acalibration phantom, as is often required in typical IGS. Thus, thepresent system does not require a grid on the c-arm to correct for thevarious types of distortion (i.e.—pin cushion, etc.). When an IGS systemis being used, the present system permits the IGS tracker to be eitherplaced at the tip of the effecter (in the case of an EM microsensor orthe like) or projected to the tip by a known offset that is more typicalof an optical system. The present system does not require any patientreference, such as a “beacon” that is standard on nearly all IGSsystems. In particular, it is not necessary to know the location of theobject's tip relative to the c-arm (the distance of the tip between theimage intensifier and the x-ray source) and the in plane movement(distance and trajectory) of the effecter

The present system and method can operate with a single image,separating metal or radio-dense material from anatomy and leaving theanatomy without the radio-dense material as a layer, or the radio-densematerial can be moved without anatomy as a layer, as depicted in FIGS.5, 6, or the layers can be moved in any combination.

The present method and system even works with distorted IGS data (likeis classically a problem with EM), as the movement won't be perfect butwill asymptotically get closer to the correct position. For instance, ifthe IGS data is inaccurate by 20%, then after the first movement, a“new” x-ray will confirm that it is 20% off. However, the system is thenrecalibrated so that now moving the new “new” image is not only moreaccurate, but the distance needed to move is only ⅕^(th) the priordistance. Thus, even if the system still has a 20% error, the nextmovement to close the gap of this 20% will be only 4% off (i.e., 20% of20%). The use of relative motion and this perpetually smaller distancemoved between each x-ray allows the present system to use noisy warpedEM data for application in the OR.

In another feature, the tip of the effecter, such as effecter 10, can berepresented on the displayed x-ray image as a slug 30 shown in thescreen shot of FIG. 7. The position of the slug can correspond to theposition of the tip of the effecter relative to the anatomy and can takevarious forms, including a circle or bulls-eye and an arrow, as depictedin FIG. 7. The appearance of the slug 30 can be varied to signifydifferent conditions in the process of navigating the effecter to thedesired anatomical position. For instance, the size or configuration ofthe slug can be indicative of the degree of accuracy associated with theparticular movement. For example, the slug can be depicted as a circlewhen the accuracy is lower and an arrow when the accuracy is greater.The size of the circle can be related to the degree of accuracy for thelocation of the tip of the effecter.

The color of the slug can be also varied to indicate certain conditions,namely conditions of the C-arm or x-ray device. For example, the slugcan be green if the current position of the C-arm is within a narrowrange of its position, 2 mm for instance, when the localizing image wasacquired, and red if the current position is outside that range. Whenthe slug changes from green to red the physician can obtain a new x-rayimage to establish a new baseline and verify the actual current positionof the effecter. As long as the color of the effecter remains green thephysician can have confidence that the actual location of the effectertip corresponds to the displayed location. As an alternative to changingcolor, the slug 30 can flash if the position of the C-arm has changed.

In the case where multiple effecters are present in a surgical site, thecolor of the slug 30 can be indicative of the particular effecterassociated therewith. It should be appreciated that all of the stepsdiscussed above can be implemented for multiple effectors for accuratenavigation of the effecters to a desired position. It can be expectedthat the multiple effecters may require positioning and re-positioningduring a procedure, so methods of the present disclosure can be modifiedaccordingly to account for multiple effecters and multiple slugs.

In another embodiment, a slug 35, shown in FIG. 7, marking the locationof the tip of the effecter can include a central element 36, in the formof a dot or small circle, corresponding to the position of the tip, anda second element 37, in the form of a circle that is at least initiallyconcentrically disposed around the central element 36. The secondelement in the form of a circle can correspond to a point on theeffecter offset along the longitudinal axis of the effecter from thetip. The location of the second element or circle 37 relative to thecentral element or dot 36 provides the physician with an indication ofthe attitude of the effecter. In the depiction of FIG. 7, the offset ofthe circle 37 relative to the dot 36 indicates that the shaft of theassociated effecter extends to the left and downward in the surgicalfield.

In an alternative embodiment, a slug 35′ can include the same firstelement in the form of a dot or small circle 36′ depicting the positionof the effecter tip, as shown in FIG. 7. However, rather than include acircle for the second element, the second element of the slug 35′ is an“I” that not only indicates the orientation of the effecter relative toits tip, but also indicates the rotation about the axis of the effecter.The angular offset of the “I” from a vertical orientation provides thesurgeon with a visual indication of the rotational orientation of theimplant, tool or instrument.

As discussed above, the present systems and methods utilize trackinginformation from a localizer system that acquires the position of theeffecter. Typical localizer systems utilize an array of optical sensorsto track an optical tracking component mounted to the end of theeffecter. This arrangement is cumbersome and often interferes with thesurgeon's field of view of the surgical site. In one aspect of thepresent disclosure, an effecter 40 includes a handle 41 with anelongated shaft 42 terminating in a working tip 43, as depicted in FIG.8. The shaft 42 is provided with optically trackable markers 44 a, 44 bin the form of optical bands that encircle the shaft so that the markersare visible at all rotational angles of the effecter. The bands may beformed by optical tape applied to the effecter or may be applieddirectly to the material of the effecter, such as by etching. The twomarkers 44 a, 44 b permit tracking the movement of the effecter in fivedegrees of freedom —X, Y, Z, pitch (X rotation) and yaw (Y rotation).The markers 44 a, 44 b are provided at a predetermined distance from theworking tip 43 so that the localizer software can use the detectedlocation of the two markers to extrapolate the 5 DOF position of theworking tip.

In one aspect of this feature of the invention, the markers 44 a, 44 bare separated by a predetermined spacing in which the spacing isindicative of the type of effecter. For instance, one spacing of themarkers may denote a cage inserter while another different spacing ofthe markers may denote a distracter. The localizer system can beconfigured to discern the spacing of the markers 44 a, 44 b and thenrefer to a stored data base to determine the nature of the effecterbeing detected. The data base includes information locating the workingtip in relation to the markers so that the position of the working tipcan be accurately determined by sensing the location of the markers. Thedata base may also include a model of the instrument that can be used togenerate the metal mask 20 described above. Once the particular effecteris identified, the localizer system will always know where the workingtip is located even when one of the two markers is obscured.

In another aspect, the markers are incorporated into a tracking element45 that can be mounted to the shaft 42′ of a tool 40′ that is otherwisesimilar to the tool 40, as shown in FIG. 9. The tacking element includesa cylindrical or partially cylindrical body 46 that can be clipped ontothe shaft 42′ and held in position with a friction grip. The cylindricalbody 46 includes the two markers 44 a′, 44 b′ in the form of bands thatencircle the body. A third marker 44 c′ can be provided on an arm 48that projects from the cylindrical body, with the third markerconstituting an optically detectable band. The addition of the thirdmarker 44 c′ adds a sixth degree of freedom to the position datadetected by the localizer device, namely roll or rotation about theZ-axis or longitudinal axis of the shaft 42′. The bands 44 a′, 44 b′ canbe spaced apart in the manner described above to denote a particulareffecter.

In an alternative embodiment, an effecter 40″ shown in FIG. 10 includesan existing conventional fiducial marker 44 a″ on the shaft 42″ of theeffecter. A tracking element 45″ includes a cylindrical or partiallycylindrical body 46″ configured to be clamped onto the shaft 42″ of theeffecter. The body 46″ includes a second marker 44 b″ in the form of aband that encircles the cylindrical body, and may include a third marker44 c″ on a perpendicular extension 48″. The two markers 44 b″, 44 c″ onthe tracking element 45″ cooperate with the existing fiducial 44 a″ onthe effecter to permit detecting the position of the effecter, andtherefore the working tip 43″, in six degrees of freedom. In thisembodiment, the tracking element 45″ is clamped to the shaft 42″ at aparticular height h relative to the working tip 43″. The height hproduces a predetermined spacing relative existing fiducial 44 a″, whichspacing can be used to identify the nature of the particular effecter. Acalibration tool may be used to position the tracking element 45″ at theproper height for a particular effecter.

As mentioned, the location of the markers on the effecter can be used toidentify the nature of the effecter—i.e., as a tool, instrument, implantetc. The imaging software remembers what effecters are in the surgicalfield as well as the positions as they are moved within that field. Evenif one of more of the markers are temporarily blocked from view of thelocalizer or tracking device, the imaging software can extrapolate theposition of the effecter based on the position of the available markers.

In a further aspect of the invention, the image processing software canbe configured to automate certain features of the system based on thetype of effecter detected and the nature of the procedure. The softwarecan permit the surgeon to identify the nature of the surgical procedure,and then this information together with the information regarding theeffecter or effecters in use can be used to toggle certain displayfeatures. The toggled features can include metal enhancement (asdiscussed herein), the nature of the slugs displayed on the x-ray image,or the use of one or two adjacent views (such as AP and lateral at thesame time).

The system described above provides a method for tracking an effecter,such as a tool T within a displayed field F, as illustrated in FIG. 11.The present disclosure further contemplates imaging software implementedby the image processing device 22 (FIG. 1) that is activated only whenthe tracked radio-dense object, such as tool T, enters the surgicalfield F and a new image has been taken by the radiologist or surgeon.When these two conditions occur, an object mask for the tool, such asthe green mask 20, is displayed and the image may be manipulated by thesurgeon based on manipulations of the effecter or other radio-denseobject. The software remains activated until a new image is taken thatdoes not include the tracked instrument. If the radio-dense objectreappears in the field F, the software remembers the original locationof the field and the tool and allows manipulation by the radiologist orsurgeon.

The software of the present disclosure thus provides a metalidentification feature that is always running in the background of theimaging software execution. The software automatically identifies thepresence of a radio-dense object in the surgical field without anyoperator intervention, and displays an image of the radio-dense objectwithout operator intervention. The present disclosure thus contemplatesa system for identifying a radio-dense object in an image field andenhancing the display of that object for the benefit of the surgeonattempting to navigate the object within the surgical field. Thesoftware disclosed herein thus identifies the nature and parameters ofthe radio-dense object without any input or intervention from theradiologist or surgeon. The software analyzes the x-ray image to locatethe radio-dense object or objects and then create a mask correspondingto the configuration of the object. When the object is moved, thesoftware can move only the object mask without modifying the underlyingimage of the surgical field. In one approach, the software utilizesexisting tracking data for the guided surgical tool to identify theregion of the image field in which the tip of the instrument or tool canbe found, and/or a general angle of projection of the tool on the x-rayobtained from the existing tracking data. The present disclosure thusprovides a system that can locate a tool T even where the tracking dataonly identifies a region R within the viewing field F (FIG. 11).

Once the radio-dense object is located, the software and system of thepresent disclosure enhances or intensifies the image of the radio-denseobject. As shown in FIG. 12A, some radio-dense objects M are difficultto see in a low dose image. As shown FIG. 12C, the problem isexacerbated when the low dose image is merged with a prior standard doseimage (FIG. 12B), such as according to the techniques described U.S.Pat. No. 8,526,700, which issued on September 3, 2013, the entiredisclosure of which is incorporated herein by reference. The presentdisclosure contemplates software executed by the image processing device122 (FIG. 1) that identifies the location of the radio-dense object(s)M, even in an image field as shown in FIG. 12A, and then intensifies theradio-dense objects M′ in a composite image shown in FIG. 12C so thatthe radio-dense objects are sufficiently visible to the surgeon. Thesoftware can locate the radio-dense object directly from the image FIG.12A, or can use angle of projection and/or location data provided by animage guidance component, to speed up the process of identifying thelocation of the radio-dense object(s) M. The system and softwaredisclosed herein thus provides means for locating and enhancingincredibly faint objects within the viewing field, even when the imageis a low dose image. Once the radio-dense object(s) M′ are located andenhanced, only the enhanced radio-dense object is moved while theunderlying baseline or composite x-ray image can remain stationary sinceonly the object is being tracked. It is further contemplated that thetracked objects can be limited to only select ones of the radio-denseobjects that may appear in a particular field of view. The non-trackedradio-dense objects can remain un-enhanced and left stationary even asthe image moves with the tracked radio-dense objects M′. Moreover, anyone or multiples of radio-dense objects in an image can be identified,enhanced and moved independently as independent masks overlying abaseline or composite x-ray image. With this feature, multiplephysicians can work simultaneously and together to position radio-denseobjects necessary for the surgical procedure, all working from the sameunderlying stationary baseline or composite x-ray image.

The system and software of the present disclosure allows isolation of aradio-dense object within an image, such as the image FIG. 13A and theisolated image in FIG. 13B. The isolated image can be used to guidemovement of the radio-dense object which can then be reintegrated withthe x-ray image at a new location as shown in FIG. 13C. This process canbe performed with any radio-dense object, once it has been identified,as illustrated in FIGS. 13D-F. The radio-dense objects can berepresented by a mask, with the masks for multiple objects beingcolor-coded, as shown in FIG. 13F.

FIGS. 14A-F shows a series of screen shots of displays generated by thepresent system and software. The first image in FIG. 14A, shows a faintobject M in a low radiation image. It is apparent from this image thatthe radio-dense object M is too faint for a surgeon to reliablymanipulate the instrument or tool. In the composite image of FIG. 14Bthe radio-dense object is even fainter. FIG. 14C shows an image of onestep in the metal identification algorithm implemented by the softwareof the present disclosure which relies on identifying linear edges thatare indicative of a non-anatomic feature. When tracking information forthe particular effecter or object is added, as shown in FIG. 14D, thecorrect linear edge is identified as the radio-dense object, which isthen enhanced and displayed in the image of FIG. 14E.

The system and software further provides two ways to view movement of atracked radio-dense object within a surgical field. The system describedin U.S. Pat. No. 8,526,700, incorporated by reference above, provides asystem for orienting a view as the x-ray device or C-arm is angled, asdepicted in FIG. 15A. In this system, when the C-arm is moved fromposition 1 to position 2, the displayed images move from the position inFIG. 15B to the position shown in FIG. 15C. In FIG. 15D, grid lines areadded that can ultimately be used to orient the C-arm to a perfectalignment for a Ferguson (flat endplate) view of a spinal field from theorthogonal x-ray image. The grid lines are parallel to the orientationof the effecter or radio-dense object.

In accordance with the present disclosure, when the radio-dense effecteror tool is moved, as shown in FIGS. 16-17, the tracked object controlsthe angle of the displayed image. The tracked object shown in FIG. 16Ais maintained in a constant orientation (such as vertical in FIG. 16B)and the x-ray image itself is rotated commensurate with the movement ofthe tracked object, as shown in FIG. 16C. It can be appreciated that thechange in angular orientation of the image between FIGS. 16b and FIG.16C is the same as the change in angular orientation of the effecterfrom position 1 to position 2 in FIG. 16A.

As an adjunct to this feature, the image data for the rotated image ofFIG. 16C can be used to identify a movement for the c-arm to produce adesired shot of the effecter and the surgical site. For instance, theimage data can be used to identify a movement angle for the c-arm togenerate an en face view down the shaft of the effecter. Similarly, theimage data can be used to center the c-arm over the shaft of theeffecter or angle the c-arm to shoot perpendicular to the effecter shaftand centered over the tip of the instrument.

Alternatively, as shown in FIGS. 17A-C, the x-ray image can remainstationary while the image or mask of the tracked radio-dense object ismoved commensurate with the actual movement of the tracked radio-denseobject. The depth of the radio-dense object can be further adjusted bymoving the metal mask or image axially along its length. The grid linescan be added to the displays, whether the tracked object remainsstationary in the field of view, as in FIGS. 18A-C, or the x-ray viewremains stationary and the image or mask of the effecter is moved, as inFIGS. 19A-C.

As described above, the imaging software of the present systemimplements a method to detect the presence and location of trackedradio-dense objects and enhances the objects. The position andorientation of the radio-dense effecter, such as a tool or instrument,in space with respect to an X-ray device are measured by a tracker orlocalizer system associated with the effecter. This tracking informationis used to translate an X-ray image of the effecter on the viewingscreen that predicts where the effecter would appear if another X-rayimage were acquired. The image of the tool can be merged with apreviously acquired image of the patient's anatomy, with the previouslyacquired image remaining static. The resulting merged image informs thephysician about the placement of the effecter relative to the anatomy.

One problem with this approach is that certain commonly used surgicaltools T can be difficult to see in an X-ray image, especially if thisimage was acquired at a low X-ray dosage, as depicted in the screen shotimages of FIG. 20. The visibility of the surgical tool is furtherdiminished by the merging of a baseline image with a subsequent low doseimage. Consequently, the present disclosure contemplates a method forenhancing the visibility of a tracked surgical tool in a merged X-rayimage.

The steps of one method implemented by the imaging software are shown inthe chart of FIG. 21. Several parameters are available to optimize themethod for particular classes of surgical tools. All steps have beendesigned to be straightforward to implement on a graphics processingunit, such as the GPU of the image processing device 122 (FIG. 1), whichperforms optimally when the same computational operation can beperformed at all pixels in an image simultaneously. In the presentimplementation, the entire operation can be applied to a standard sizeimage in half a second with a consumer-grade graphics card, whichsuffices for most usage patterns.

One step of the method is to detect rectangles within the x-ray image.Each pixel is assigned a score that represents how well a darkrectangular pattern can be fitted to the neighborhood centered on thepixel. A rectangle is defined by its angle, width, and length. The scorefor a particular rectangle is the sum of the differences in theintensity values between points along the inside of the long edges ofthe rectangle and points along the outside (FIG. 22). This scorecalculation is performed for many different possible rectangles over arange of angles, widths, and lengths, and the highest score is reported,along with the corresponding angle, width, and length.

When tracking a metal tool that is especially thick, the differencecalculation can also be performed at multiple depths in the interior ofthe rectangle. This ensures that the rectangle has a homogeneousinterior. The intensity difference formula can be clamped to a narrowrange of possible values, and scaled by a fractional exponent, so thatespecially large intensity differences will not have a disproportionateinfluence on the final score.

In a next step, pixels of the x-ray image are assigned to therectangles. This step extends the results from rectangle detection. Foreach pixel, the neighborhood around the pixel is searched for thehighest-scoring rectangle that overlaps it (FIG. 23). This score isreported, along with the corresponding angle, width, and length. Thisstep is needed because rectangles have corners and intersections, andthe pixels at these locations are not centered on the rectangle thatbest contains them.

In an X-ray image, a surgical tool may comprise multiple connectedrectangles, so it is preferable to join the multiple rectangles togetherinto a single contiguous region. In order to determine whether or notpixels belong to the same region, for two adjacent pixels, each of whichhas been assigned a rectangle score, angle, width, and length from theprevious steps, the connection criterion is the sum of the differencesin the rectangle scores, angles, widths, and lengths (FIG. 24). If theconnection criterion falls below a threshold, the pixels share aconnection. The relative contributions of the scores, angle, widths, andlengths can be weighted in order to control their influence on thecriterion. Each pixel has 8 neighbors to which it might potentially beconnected. This operation is performed at each pixel for all 8directions. To reduce computation time, connections between pixels withvery low rectangle scores can be ignored.

In the next step the tracking information obtained from the localizer ortracking device for the tool is related to the pixels. The trackingdevice provides data for the position and orientation of the tip of thesurgical tool in space. This tip can be virtually projected onto thesurface of the X-ray camera and related to a point and an angle withinthe X-ray image, as described above. For enhancement purposes, theprimary interest is in rectangular image features that have a positionand angle that are close to the projected tool tip. For each pixel, thedistance to the projected tool tip is calculated, and the differencebetween the angle of the tool tip and the angle of the rectangle at thepixel is calculated. These values can be clamped and scaled with anexponent to yield weights that quantify the spatial proximity andangular proximity of the pixel to the tool tip (FIG. 25). A tool istypically a long thin object, and pixels behind the tip belong to theobject while pixels in front of the tip do not. This prior knowledge canbe encoded by including orientation information into the calculation ofspatial proximity.

The pixels are then grouped into contiguous regions. Each region willhave a unique index, a rectangle score, a spatial proximity, and anangle proximity. These values will be accessible at each pixel in theregion. There are various algorithms available for this task. Thealgorithm used here was chosen because it can be performed at each pixelin parallel. The region growing algorithm proceeds iteratively. At eachiteration, for each of 8 possible directions, each pixel looks at itsneighbor in that direction. If the pixel shares a connection with itsneighbor, then they compare rectangle scores. If the neighbor has ahigher score, then the pixel receives the score and the index of itsneighbor. Otherwise, if the scores are equal, and the neighbor has ahigher index, then the pixel receives the index of its neighbor. If thepixel shares a connection with its neighbor and the neighbor has ahigher spatial proximity, then the pixel receives the spatial proximityof its neighbor. If the pixel shares a connection with its neighbor andthe neighbor has a higher angular proximity, then the pixel receives theangular proximity of its neighbor. At the end of the iteration, if theindex, score, spatial proximity or angular proximity have changed forany pixel in the image, then another iteration is performed. Otherwise,the algorithm halts.

When the algorithm has finished, each pixel has been assigned to aregion. Each region has a unique index, and each region has the bestrectangle score, spatial proximity, and angular proximity out of all thepixels in the region. These values are stored at each pixel in theregion.

Next, the regions are visually enhanced. In an X-ray image, a surgicaltool should appear darker than the surrounding area. To enhancevisibility, the pixels inside the region can be made darker, and thepixels outside the region lighter (FIG. 26). The changes to intensityshould be smooth so that no spurious textures are introduced into theimage, and so that the enhancement is robust in the presence ofpotential errors from the previous steps. Each pixel looks at each otherpixel in the neighborhood. The score, angle, width, and length of therectangle centered at the neighbor are found, as well as the score,spatial proximity, and angular proximity of the region to which theneighbor belongs.

The latitudinal and longitudinal axes of the neighboring rectangle aredetermined. The distance between the pixel and its neighbor is expressedas a sum of a latitudinal component and a longitudinal component. Thelatitudinal component is passed to a difference-of-Gaussians model thatreturns a negative value for pixels within the interior of the rectangleand a positive value in the exterior. The longitudinal component ispassed to a hyperbolic model that returns a fraction that approaches 0as the longitudinal distance grows. The offset to the pixel contributedby this neighbor is a product of the rectangle score, region score,spatial proximity, angular proximity, latitudinal weight, andlongitudinal weight. The offsets from all neighboring pixels are addedtogether. This step yields an intensity offset that can be used in theimage merging step.

The tracking information is then used to isolate the region of interest.The tracking information is used to weight the regions according totheir proximity to the tool tip. This will generate a mask that can beused to selectively weight different parts of the image when the imageis merged (FIG. 27). For each pixel, the mask value is the product ofthe region score, spatial proximity, and angle proximity. This value canbe thresholded and scaled with an exponent to suppress irrelevantregions of the image. The edges of the regions are often jagged and donot exactly correspond to the tool. It is thus necessary to expand theregion and smooth the boundaries so that the final merged image will nothave any visually unpleasant discontinuities. This is accomplished withmorphological dilation, followed by convolution with a Gaussian kernel.The values of the pixels in the mask are clamped to between 0 and 1. Avalue of 0 indicates that the pixel does not belong to the region ofinterest; a value of 1 indicates that the pixel fully belongs to theregion of interest.

In the next step, the entire tool image is enhanced. The intensityoffset image is added to the original image of the tool. The resultingsum may now have pixels outside the acceptable intensity range of 0 to255. To bring the intensities back to an acceptable range, and tofurther improve the contrast around the metal edges, the histogram ofthe intensities within the mask region of the image sum is constructedin order to determine low and high quantiles. All intensities in the sumare scaled linearly so that the low quantile is now 0 and the highquantile is now 255. This yields an enhanced tool image.

Finally, the enhanced tool image is added to the anatomical image. Atpixels where the mask value is high, the enhanced tool imagepredominates, while at pixels where the mask value is low, theanatomical image predominates. The maximum and minimum ratios of the twoimages are chosen so that neither image is ever completely suppressed.This final merged image is displayed to the user as depicted in thescreen shot of FIG. 28.

In one aspect of the present invention, the effecter tracking featuredescribed above is used in connection with a system and method forproviding updated images of the surgical field and patient anatomywithout the requirement for full dose imaging. The image processingdevice 122 is thus further configured to provide high quality real-timeimages on the displays 123, 124 that are derived from lower detailimages obtained using lower doses (LD) of radiation. By way of example,FIG. 29A is a “full dose” (FD) x-ray image, while FIG. 29B is a low doseand/or pulsed (LD) image of the same anatomy. It is apparent that the LDimage is too “noisy” and does not provide enough information about thelocal anatomy for accurate image guided surgery. While the FD imageprovides a crisp view of the surgical site, the higher radiation dosemakes taking multiple FD images during a procedure highly problematic.Using the steps described herein, the surgeon is provided with a currentimage shown in FIG. 29C that significantly reduces the noise of the LDimage, in some cases by about 90%, so that surgeon is provided with aclear real-time image using a pulsed or low dose radiation setting. Thiscapability allows for dramatically less radiation exposure during theimaging to verify the position of instruments and implants during theprocedure.

The flowchart of FIG. 30 illustrates steps of the method for generatingthe current image shown in FIG. 29C. In a first step 200, a baselinehigh resolution FD image is acquired of the surgical site and stored ina memory associated with the image processing device. In some caseswhere the C-arm is moved during the procedure, multiple high resolutionimages can be obtained at different locations in the surgical site, andthen these multiple images “stitched” together to form a composite baseimage using known image stitching techniques). Movement of the C-arm,and more particularly “tracking” the acquired image during thesemovements, is accounted for in other steps described in more detailherein. For the present discussion it is assumed that the imaging systemis relatively fixed, meaning that only very limited movement of theC-arm and/or patient are contemplated, such as might arise in anepidural pain procedure, spinal K-wire placement or stone extraction.The baseline image is projected in step 202 on the display 123 forverification that the surgical site is properly centered within theimage. In some cases, new FD images may be obtained until a suitablebaseline image is obtained. In procedures in which the C-arm is moved,new baseline images are obtained at the new location of the imagingdevice, as discussed below. If the displayed image is acceptable as abaseline image, a button may be depressed on a user interface, such ason the display device 126 or interface 125. In procedures performed onanatomical regions where a substantial amount of motion due tophysiological processes (such as respiration) is expected, multiplebaseline images may be acquired for the same region over multiple phasesof the cycle. These images may be tagged to temporal data from othermedical instruments, such as an ECG or pulse oximeter.

Once the baseline image is acquired, a baseline image set is generatedin step 204 in which the original baseline image is digitally rotated,translated and resized to create thousands of permutations of theoriginal baseline image. For instance, a typical two dimensional (2D)image of 128×128 pixels may be translated ±15 pixels in the x and ydirections at 1 pixel intervals, rotated ±9° at 3° intervals and scaledfrom 92.5% to 107.5% at 2.5% intervals (4 degrees of freedom, 4D),yielding 47,089 images in the baseline image set. (A three-dimensional(3D) image will imply a 6D solution space due to the addition of twoadditional rotations orthogonal to the x and y axis. An original CTimage data set can be used to form many thousands of DRRs in a similarfashion.) Thus, in this step, the original baseline image spawnsthousands of new image representations as if the original baseline imagewas acquired at each of the different movement permutations. This“solution space” may be stored in a graphics card memory, such as in thegraphics processing unit (GPU) of the image processing device 122, instep 206 or formed as a new image which is then sent to the GPU,depending on the number of images in the solution space and the speed atwhich the GPU can produce those images. With current computing power, ona free standing, medical grade computer, the generation of a baselineimage set having nearly 850,000 images can occur in less than one secondin a GPU because the multiple processors of the GPU can eachsimultaneously process an image.

During the procedure, a new LD image is acquired in step 208, stored inthe memory associated with the image processing device, and projected ondisplay 123. Since the new image is obtained at a lower dose ofradiation it is very noisy. The present invention thus provides stepsfor “merging” the new image with an image from the baseline image set toproduce a clearer image on the second display 124 that conveys moreuseful information to the surgeon. The invention thus contemplates animage recognition or registration step 210 in which the new image iscompared to the images in the baseline image set to find a statisticallymeaningful match. A new “merged” image is generated in step 212 that maybe displayed on display 124 adjacent the view of the original new image.At various times throughout the procedure, a new baseline image may beobtained in step 216 that is used to generate a new baseline image setin step 204.

Step 210 contemplates comparing the current new image to the images inthe baseline image set. Since this step occurs during the surgicalprocedure, time and accuracy are critical. Preferably, the step canobtain an image registration in less than one second so that there is nomeaningful delay between when the image is taken by the C-arm and whenthe merged image is displayed on the device 126. Various algorithms maybe employed that may be dependent on various factors, such as the numberof images in the baseline image set, the size and speed of the computerprocessor or graphics processor performing the algorithm calculations,the time allotted to perform the computations, and the size of theimages being compared (e.g., 128×128 pixels, 1024×1024 pixels, etc). Inone approach, comparisons are made between pixels at predeterminedlocations described above in a grid pattern throughout 4D space. Inanother heuristic approach, pixel comparisons can be concentrated inregions of the images believed to provide a greater likelihood of arelevant match. In yet another approach, a principal component analysis(PCA) is performed, which can allow for comparison to a larger number oflarger images in the allotted amount of time than is permitted with thefull resolution grid approach. Further details of these approaches aredisclosed in U.S. Pat. No. 8,526,700, incorporated by reference above.

In the image guided surgical procedures, tools, implants and instrumentswill inevitably appear in the image field. These objects are typicallyradiodense and consequently block the relevant patient anatomy fromview. The new image obtained in step 210 will thus include an artifactof the tool T that will not correlate to any of the baseline image set.The image registration steps may be modified to account for the toolartifacts on the new image. In one approach, the new image may beevaluated to determine the number of image pixels that are “blocked” bythe tool. In another approach, the image recognition or registrationstep 210 may include steps to measure the similarity of the LD image toa transformed version of the baseline image (i.e., a baseline image thathas been transformed to account for movement of the C-arm, as describedbelow relative to FIG. 34) or of the patient. Further details of theseapproaches are disclosed in U.S. Pat. No. 8,526,700, incorporated byreference above.

As previously explained, non-anatomical features may be present in theimage, such as radio-dense effecters in the form of tool, instruments orimplants. The effecters may be tracked according to the processesdescribed above. During a surgical procedure it is still desirable todisplay an image of the entire surgical site, including of anatomy thatis blocked by the radio-dense effecter. Thus, in a further aspect of theimage manipulation steps, a mask image can be generated that identifieswhether or not a pixel is part of an anatomical feature. Once thenon-anatomical features are obtained, the baseline image of the anatomyobscured by the non-anatomical features can be merged into the image toshow the surgical site without the radio-dense effecter.

In one aspect, an anatomical pixel may be assigned a value of 1 while anon-anatomical pixel is assigned a value of 0. This assignment of valuesallows both the baseline image and the LD image to be multiplied by thecorresponding mask images before the similarity function is computed asdescribed above In other words, the mask image can eliminate thenon-anatomical pixels to avoid any impact on the similarity functioncalculations. To determine whether or not a pixel is anatomical, avariety of functions can be calculated in the neighborhood around eachpixel. These functions of the neighborhood may include the standarddeviation, the magnitude of the gradient, and/or the correspondingvalues of the pixel in the original grayscale image and in the filteredimage. The “neighborhood” around a pixel includes a pre-determinednumber of adjacent pixels, such as a 5×5 or a 3×3 grid. Additionally,these functions can be compounded, for example, by finding the standarddeviation of the neighborhood of the standard deviations, or bycomputing a quadratic function of the standard deviation and themagnitude of the gradient. One example of a suitable function of theneighborhood is the use of edge detection techniques to distinguishbetween bone and radio-dense instruments. Metal presents a “sharper”edge than bone and this difference can be determined using standarddeviation or gradient calculations in the neighborhood of an “edge”pixel. The neighborhood functions may thus determine whether a pixel isanatomic or non-anatomic based on this edge detection approach andassign a value of 1 or 0 as appropriate to the pixel.

Once a set of values has been computed for the particular pixel, thevalues can be compared against thresholds determined from measurementsof previously-acquired images and a binary value can be assigned to thepixel based on the number of thresholds that are exceeded.Alternatively, a fractional value between 0 and 1 may be assigned to thepixel, reflecting a degree of certainty about the identity of the pixelas part of an anatomic or non-anatomic feature. These steps can beaccelerated with a GPU by assigning the computations at one pixel in theimage to one processor on the GPU, thereby enabling values for multiplepixels to be computed simultaneously. The masks can be manipulated tofill in and expand regions that correspond to non-anatomical featuresusing combinations of morphological image operations such as erosion anddilation.

An example of the steps of this approach is illustrated in the images ofFIGS. 31A-31P. In FIG. 31A, an image of a surgical site includesanatomic features (the patient's skull) and non-anatomic features (suchas a clamp). The image of FIG. 31A is filtered for edge enhancement toproduce the filtered image of FIG. 31B. It can be appreciated that thisimage is represented by thousands of pixels in a conventional manner,with the intensity value of each pixel modified according to the edgeenhancement attributes of the filter. In this example, the filter is aButterworth filter. This filtered image is then subject to eightdifferent techniques for generating a mask corresponding to thenon-anatomic features. Thus, the neighborhood functions described inU.S. Pat. No. 8,526,700 (namely, standard deviation, gradient andcompounded functions thereof) are applied to the filtered image FIG. 31Bto produce different images FIGS. 31C-31J. Each of these images isstored as a baseline image for comparison to and registration with alive LD image.

Each of the images of FIGS. 31C-31J is used to generate a mask. The maskgeneration process may be by comparison of the pixel intensities to athreshold value or by a lookup table in which intensity valuescorresponding to known non-anatomic features is compared to the pixelintensity. The masks generated by the threshold and lookup tabletechniques for one of the neighborhood function images is shown in FIGS.31K-31L. The masks can then be manipulated to fill in and expand regionsthat correspond to the non-anatomical features, as represented in theimages of FIGS. 31M-31N. The resulting mask is then applied to thefiltered image of FIG. 31B to produce the “final” baseline images ofFIGS. 310-31P that will be compared to the live LD image. As explainedabove, each of these calculations and pixel evaluations can be performedin the individual processors of the GPU so that all of these images canbe generated in an extremely short time. Moreover, each of these maskedbaseline images can be transformed to account for movement of thesurgical field or imaging device and compared to the live LD image tofind the baseline image that yields the highest Z score corresponding tothe best alignment between baseline and LD images. This selectedbaseline image is then used in manner explained below.

Once the image registration is complete, the new image may be displayedwith the selected image from the baseline image set in different ways.In one approach, the two images are merged, as illustrated in FIGS. 32A,B. The original new image is shown in FIG. 32A with the instrument Tplainly visible and blocking the underlying anatomy. A partially mergedimage generated in step 212 (FIG. 30) is shown in FIG. 32B in which theinstrument T is still visible but substantially mitigated and theunderlying anatomy is visible. The two images may be merged by combiningthe digital representation of the images in a conventional manner, suchas by adding or averaging pixel data for the two images. In oneembodiment, the surgeon may identify one or more specific regions ofinterest in the displayed image, such as through the user interface 125,and the merging operation can be configured to utilize the baselineimage data for the display outside the region of interest and conductthe merging operation for the display within the region of interest. Theuser interface 125 may be provided with a “slider” that controls theamount the baseline image versus the new image that is displayed in themerged image.

As described in U.S. Pat. No. 8,526,700, an image enhancement system canbe used to minimize radio-opaque instruments and allow visualization ofanatomy underlying the instrumentation. Alternatively, the system can beoperable to enhance selected instrumentation in an image or collectionof images. In particular, the masks describe above used to identify thelocation of the non-anatomic features can be selectively enhanced in animage. The same data can also be alternately manipulated to enhance theanatomic features and the selected instrumentation. This feature can beused to allow the surgeon to confirm that the visualized landscape looksas expected, to help identify possible distortions in the image, and toassist in image guided instrumentation procedures. Since the bone screwis radio-opaque it can be easily visualized under a very low dose x-raya low dose new image can be used to identify the location of theinstrumentation while merged with the high dose baseline anatomy image.Multiple very low dose images can be acquired as the bone screw isadvanced into the bone to verify the proper positioning of the bonescrew. Since the geometry of the instrument, such as the bone screw, isknown (or can be obtained or derived such as from image guidance, 2-Dprojection or both), the pixel data used to represent the instrument inthe x-ray image can be replaced with a CAD model mapped onto the edgeenhanced image of the instrument.

As indicated above, the present invention also contemplates a surgicalnavigation procedure in which the imaging device or C-arm 103 is moved.The position of the C-arm can be tracked, rather than or in addition totracking the position of the surgical instruments and implants, usingcommercially available tracking devices or the DICOM information fromthe imaging device. Tracking the C-arm requires a degree of accuracythat is much less than the accuracy required to track the instrumentsand implants. In this embodiment, the image processing device 122receives tracking information from the tracking device 130. Tracking theposition of the C-arm can account for “drift”, which is a gradualmisalignment of the physical space and the imaging (or virtual) space.This “drift” can occur because of subtle patient movements, inadvertentcontact with the table or imaging device and even gravity. Thismisalignment is often visually imperceptible, but can generatenoticeable shifts in the image viewed by the surgeon. These shifts canbe problematic when the surgical navigation procedure is being performed(and a physician is relying on the information obtained from thisdevice) or when alignment of new to baseline images is required toimprove image clarity. The use of image processing eliminates theinevitable misalignment of baseline and new images. The image processingdevice 122 further may incorporate a calibration mode in which thecurrent image of the anatomy is compared to the predicted image. Thedifference between the predicted and actual movement of the image can beaccounted for by an inaccurate knowledge of the “center of mass” or COM,described below, and drift. Once a few images are obtained and the COMis accurately established, recalibration of the system can occurautomatically with each successive image taken and thereby eliminatingthe impact of drift.

A display with two view finder images can be utilized by the radiologytechnician to orient the C-arm to acquire a new image at the sameorientation as a baseline image. In this embodiment, the two view finderimages are orthogonal images, such as an anterior-posterior (AP) image(passing through the body from front to back) and a lateral (LAT) image(passing through the body shoulder to shoulder). The technician seeks toalign both view finder images to corresponding AP and LAT baselineimages. As the C-arm is moved by the technician, both images are trackedsimultaneously, similar to the single view finder described above. Itcan be appreciated that the two view navigation images may be derivedfrom a baseline image and a single shot or X-ray image at a currentposition, such as a single AP image. As the view finder for the AP imageis moved to position the view at a desired location, the second viewfinder image displays the projection of that image in the orthogonalplane (i.e., the lateral view). The physician and x-ray technician canthus maneuver the C-arm to the desired location for a lateral view basedon the projection of the original AP view. Once the C-arm is alignedwith the desired location, the C-arm can then actually be positioned toobtain the orthogonal (i.e., lateral) x-ray image.

The present invention can also be used with a feature that enhances thecommunication between the surgeon and the radiology technician. Duringthe course of a procedure the surgeon may request images at particularlocations or orientations. One example is what is known as a “Fergusonview” in spinal procedures in which an AP oriented C-arm is canted toalign directly over a vertebral end plate with the end plate oriented“flat” or essentially parallel with the beam axis of the C-arm.Obtaining a Ferguson view requires rotating the C-arm or the patienttable while obtaining multiple AP views of the spine, which iscumbersome and inaccurate using current techniques, requiring a numberof fluoroscopic images to be performed to find the one best aligned tothe endplate. The present invention allows the surgeon to overlay a gridonto a single image or stitched image and provide labels for anatomicfeatures that can then be used by the technician to orient the C-arm.Thus, as shown in FIG. 33A, the image processing device 122 isconfigured to allow the surgeon to place a grid 245 within the trackingcircle 240 overlaid onto a Lateral image. The surgeon may also locatelabels 250 identifying anatomic structure, in this case spinalvertebrae. In this particular example, the goal is to align the L2-L3disc space with the center grid line 246. To assist the technician, atrajectory arrow 255 is overlaid onto the image to indicate thetrajectory of an image acquired with the C-arm in the current position.As the C-arm moves, changing orientation off of pure AP, the imageprocessing device evaluates the C-arm position data obtained from thetracking device 230 to determine the new orientation for trajectoryarrow 255. The trajectory arrow thus moves with the C-arm so that whenit is aligned with the center grid line 246, as shown in FIG. 33B, thetechnician can shoot the image knowing that the C-arm is properlyaligned to obtain a Ferguson view along the L3 endplate. Thus,monitoring the lateral view until it is rotated and centered along thecenter grid line allows the radiology technician to find the AP Fergusonangle without guessing and taking a number of incorrect images.

In another feature, a radiodense asymmetric shape or glyph can be placedin a known location on the C-arm detector. This creates the ability tolink the coordinate frame of the C-arm to the arbitrary orientation ofthe C-arm's image coordinate frame. As the C-arm's display may bemodified to generate an image having any rotation or mirroring,detecting this shape radically simplifies the process of imagecomparison and image stitching. Thus, as shown in FIG. 34, the baselineimage B includes the indicia or glyph “K” at the 9 o'clock position ofthe image. In an alternative embodiment, the glyph may be in the form ofan array of radio-opaque beads embedded in a radio-transparent componentmounted to a C-arm collar, such as in a right triangular pattern. Sincethe physical orientation and location of the glyph relative to the C-armis fixed, knowing the location and orientation of the glyph in a 2Dimage provides an automatic indication of the orientation of the imagewith respect to the physical world. The new image N is obtained in whichthe glyph has been rotated by the physician or technologist away fromthe default orientation. Comparing this new image to the baseline imageset is unlikely to produce any registration between images due to thisangular offset. In one embodiment, the image processing device detectsthe actual rotation of the C-arm from the baseline orientation while inanother embodiment the image processing device uses image recognitionsoftware to locate the “K” glyph in the new image and determine theangular offset from the default position. This angular offset is used toalter the rotation and/or mirror image the baseline image set. Thebaseline image selected in the image registration step 210 is maintainedin its transformed orientation to be merged with the newly acquiredimage. This transformation can include rotation and mirror-imaging, toeliminate the display effect that is present on a C-arm. The rotationand mirroring can be easily verified by the orientation of the glyph inthe image. It is contemplated that the glyph, whether the “K” or theradio-opaque bead array, provides the physician with the ability tocontrol the way that the image is displayed for navigation independentof the way that the image appears on the X-ray screen used by thetechnician. In other words, the imaging and navigation system disclosedherein allows the physician to rotate, mirror or otherwise manipulatethe displayed image in a manner that physician wants to see whileperforming the procedure. The glyph provides a clear indication of themanner in which the image used by the physician has been manipulated inrelation to the X-ray image. Once the physician's desired orientation ofthe displayed image has been set, the ensuing images retain that sameorientation regardless of how the C-arm has been moved.

The image processing device configured as described herein providesthree general features that: (1) reduce the amount of radiation exposurerequired for acceptable live images, (2) provide images to the surgeonthat can facilitate the surgical procedure, and (3) improve thecommunication between the radiology technician and the surgeon. Withrespect to the aspect of reducing the radiation exposure, the presentinvention permits low dose images to be taken throughout the surgicalprocedure to verify the position of an effecter, such as tool,instrument or implant, and/or to account for movements of the C-arm. Thesystems and methods herein fill in the gaps created by “noise” in thecurrent image to produce a composite or merged image of the currentfield of view with the detail of a full dose image. In practice thisallows for highly usable, high quality images of the patient's anatomygenerated with an order of magnitude reduction in radiation exposurethan standard FD imaging using unmodified features present on allcommon, commercially available C-arms. The techniques for imageregistration described herein can be implemented in a graphic processingunit and can occur in a second or so to be truly interactive; whenrequired such as in CINE mode, image registration can occur multipletimes per second. A user interface allows the surgeon to determine thelevel of confidence required for acquiring registered image and givesthe surgeon options on the nature of the display, ranging fromside-by-side views to fade in/out merged views.

With respect to the feature of providing images to the surgeon thatfacilitate the surgical procedure, several digital imaging techniquescan be used to improve the user's experience. One example is an imagetracking feature that can be used to maintain the image displayed to thesurgeon in an essentially a “stationary” position regardless of anyposition changes that may occur between image captures. In accordancewith this feature, the baseline image can be fixed in space and newimages adjust to it rather than the converse. When successive images aretaken during a step in a procedure each new image can be stabilizedrelative to the prior images so that the particular object of interest(e.g.—anatomy or instrument) is kept stationary in successive views. Forexample, as sequential images are taken as a bone screw is introducedinto a body part, the body part remains stationary on the display screenso that the actual progress of the screw can be directly observed.

In another aspect of this feature, the current image including blockingeffecters can be compared to earlier images without any blockingeffecters. In the registration process, the image processing device cangenerate a merged image between new image and baseline image thatdeemphasizes the blocking nature of the object from the displayed image.The user interface also provides the physician with the capability tofade the blocking object in and out of the displayed view.

In other embodiments in which the effecter itself is being tracked, theimage processing device can obtain position data from a tracking devicefollowing the position of the blocking object and use that position datato either move a full image including the effecter or to determine theproper location and orientation of a virtual object in the displayedimage. The virtual object may be applied to a baseline image to becompared with a new current image to serve as a check step—if the newimage matches the generated image (both tool and anatomy) within a giventolerance then the surgery can proceed. If the match is poor, thesurgery can be stopped (in the case of automated surgery) and/orrecalibration can take place. This allows for a closed-loop feedbackfeature to facilitate the safety of automation of medical intervention.

In the third feature—improving communication—the image processing devicedescribed herein allows the surgeon to annotate an image in a mannerthat can help guide the technician in the positioning of the C-arm as tohow and where to take a new picture or help the surgeon in guiding theeffecter (tool, instrument or implant) to a desired location relative tothe patient's anatomy. The user interface 125 of the image processingdevice 122 provides a vehicle for the surgeon to add a grid to thedisplayed image, label anatomic structures and/or identify trajectoriesfor alignment of the imaging device.

The same system and techniques described above may be implemented wherea collimator is used to reduce the field of exposure of the patient. Forinstance, as shown in FIG. 35A, a collimator may be used to limit thefield of exposure to the area 300 which presumably contains the criticalanatomy to be visualized by the surgeon or medical personnel. As isapparent from FIG. 35A the collimator prevents viewing the region 301that is covered by the plates of the collimator. Using the system andmethods described above, prior images of the area 315 outside thecollimated area 300 are not visible to the surgeon in the expanded fieldof view 310 shown in FIG. 35B.

The present disclosure should be considered as illustrative and notrestrictive in character. It is understood that only certain embodimentshave been presented and that all changes, modifications and furtherapplications that come within the spirit of the disclosure are desiredto be protected.

What is claimed is:
 1. A method for generating a display of an image ofa patient's internal anatomy and of radio-dense effecter in a surgicalfield during a medical procedure, comprising: acquiring a highresolution baseline image of the surgical field including the patient'sanatomy at a baseline orientation; digitally manipulating the highresolution baseline image to produce a baseline image set includingrepresentative images of the baseline image at a plurality ofpermutations of movements of the baseline image from the baselineorientation; acquiring an image of the radio-dense effecter in thesurgical field; overlaying the image of the radio-dense effecter on thebaseline image of the surgical field with the image of the radio-denseeffecter positioned relative to the image of the patient's anatomy inthe same manner as the actual radio-dense effecter is positionedrelative to the actual anatomy; tracking the movement of the radio-denseeffecter; during movement of the radio-dense effecter displaying theoverlaid images with the image of the radio-dense effecter moving inaccordance with the tracked movement of the radio-dense effecter;acquiring a new image of the surgical field at a lower resolution;comparing the new image to the representative images in the baselineimage set and selecting the representative image having an acceptabledegree of correlation with the new image; and merging the selectedrepresentative image with the new image and displaying the merged image.2. The method of claim 1, wherein the image of the radio-dense effecteris acquired from the baseline image.
 3. The method of claim 1, wherein:the baseline image is acquired as a full dose x-ray image of thepatient's anatomy; and the image of the radio-dense effecter is acquiredfrom a less than full dose x-ray image of the patient's anatomy with theradio-dense effecter in an initial position relative to the anatomy. 4.The method of claim 1, further comprising: acquiring a new image of theeffecter in the surgical field after the effecter has been movedrelative to the anatomy; subsequently overlaying the new image of theeffecter relative to the baseline image as the radio-dense effecter issubsequently moved relative to the anatomy.
 5. The method of claim 1,wherein the step of overlaying includes overlaying the image of theradio-dense effecter on the merged image.
 6. The method of claim 1,wherein the step of acquiring an image of the radio-dense effecter inthe surgical field includes altering the image.
 7. The method of claim6, wherein the step of altering the image of the radio-dense effecter inthe surgical field includes enhancing the specific image of the effecteritself.
 8. The method of claim 6, wherein the step of altering the imageof the radio-dense effecter in the surgical field includes reducing theintensity of the specific image of the anatomy relative to the intensityof the specific image of the radio-dense effecter.
 9. The method ofclaim 6, wherein the step of altering the image of the radio-denseeffecter includes replacing the image of the actual effecter in thesurgical field with a mask of the radio-dense effecter.
 10. The methodof claim 6, wherein the step of altering the image of the radio-denseeffecter includes generating an image of a slug indicative of theworking tip of the radio-dense effecter.
 11. The method of claim 10,wherein the configuration of the image of the slug changes if theimaging device used to acquire the baseline image is moved.
 12. Themethod of claim 10, wherein: the image of the slug includes a centralelement and a second element representing a point on the radio-denseeffecter offset from the working tip along a longitudinal axis of theradio-dense effecter; and the orientation of the second element relativeto the central element is based on the angular orientation of theradio-dense effecter relative to the surgical field.
 14. The method ofclaim 12, wherein the central element is a dot or a small circle and thesecond element is a larger concentric circle.
 15. The method of claim12, wherein the central element is a dot or a small circle and thesecond element is a non-circular element adapted to represent a rotationof the radio-dense effecter about its longitudinal axis.